Papers with multi-modal representations
Progressive Visual Refinement for Multi-modal Summarization (2026.eacl-short)
Copied to clipboard
| Challenge: | Multi-modal summarization (MMS) is a critical research area driven by the proliferation of multimedia content. |
| Approach: | They propose a patch-refined visual information network to exploit multimodal information . they propose combining visual information with textual information to generate concise summaries . |
| Outcome: | Extensive experiments on two public MMS datasets show the superiority of the proposed model. |
Logographic Information Aids Learning Better Representations for Natural Language Inference (2022.findings-aacl)
Copied to clipboard
| Challenge: | Statistical language models often ignore information related to logographic features of written text . many languages, including Chinese and Vietnamese, use logographic writing systems . |
| Approach: | They propose to provide language models with logographic information to learn better semantic representations. |
| Outcome: | The proposed model improves semantic representations in six languages with logographic writing systems. |
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games (2020.coling-main)
Copied to clipboard
Alessandro Suglia, Antonio Vergari, Ioannis Konstas, Yonatan Bisk, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon
| Challenge: | Existing models fail to learn multi-modal representations, relying on category labels at inference time. |
| Approach: | They propose a "imagination" module that learns context-aware and category-awful latent embeddings without relying on category labels at inference time. |
| Outcome: | The imagination module outperforms state-of-the-art competitors by 8.26% gameplay accuracy in the CompGuessWhat?! benchmark. |
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing vision-and-language pretraining approaches rely on external object detectors to encode images in a multi-modal transformer framework. |
| Approach: | They propose an object-aware end-to-end VLP framework which feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly. |
| Outcome: | The proposed framework achieves competitive or superior performances on vision-language tasks. |
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning (N18-2)
Copied to clipboard
| Challenge: | Existing multi-modal fusion methods have shown encouraging results in video understanding, but how to selectively fuse the multi-dimensional representations at different levels of details remains unexplored. |
| Approach: | They propose a hierarchically aligned cross-modal attention framework to fuse audio and visual cues at different levels of detail. |
| Outcome: | The proposed framework outperforms the previous best systems on the video captioning task. |
AHVE-CNER: Aligned Hanzi Visual Encoding Enhance Chinese Named Entity Recognition with Multi-Information (2025.coling-main)
Copied to clipboard
| Challenge: | Existing glyph-based models neglect the relationship between pictorial elements and radicals for Named Entity Recognition (NER) tasks. |
| Approach: | They propose a model that integrates multi-source visual and phonetic information of Hanzi . they propose combining pictographic features with radicals to facilitate integration . |
| Outcome: | The proposed model improves performance on benchmark datasets. |
Modeling Coreference Relations in Visual Dialog (2021.eacl-main)
Copied to clipboard
| Challenge: | Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image. |
| Approach: | They propose two soft constraints that can improve the model’s ability of resolving coreferences in dialog in an unsupervised way based on linguistic knowledge and discourse features of human dialog. |
| Outcome: | The proposed model achieves state-of-the-art performance on the VisDial v1.0 dataset without pretraining on other vision language datasets. |
CateEA: Enhancing Entity Alignment via Implicit Category Supervision (2025.coling-main)
Copied to clipboard
| Challenge: | Existing Entity Alignment methods neglect the inherent semantic information of entities, limiting alignment precision and robustness. |
| Approach: | They propose to combine implicit category information into multi-modal representations by generating pseudo-category labels from entity embeddings and integrating them into a multi-task learning framework. |
| Outcome: | Experiments on benchmark datasets show that CateEA outperforms state-of-the-art methods in various settings. |
Decompose, Prioritize, and Eliminate: Dynamically Integrating Diverse Representations for Multimodal Named Entity Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research on multi-modal Named Entity Recognition (MNER) does not integrate all multi-modal representations to provide rich contextual information to improve NER. |
| Approach: | They propose an iterative reasoning framework that integrates all the diverse multi-modal representations following the strategy of "decompose, prioritize, and eliminate" . they propose to use hierarchically connected fusion layers to prioritize transitions from "easy-to-hard" and "coarse-to fine" |
| Outcome: | The proposed framework integrates all the diverse multi-modal representations following the strategy of "decompose, prioritize, and eliminate". |
LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained models on document images with transformer-based backbones have led to significant performance gains in this field. |
| Approach: | They propose a multi-modal pre-training model that combines text, layout and image . they propose to use local 1D position instead of global 1D positions as layout input . |
| Outcome: | The proposed model can achieve state-of-the-art results on a wide variety of VrDU problems. |
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks. |
| Approach: | They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks. |
| Outcome: | The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities. |